Keyframe-based visual–inertial odometry using nonlinear optimization

Combining visual and inertial measurements has become popular in mobile robotics, since the two sensing modalities offer complementary characteristics that make them the ideal choice for accurate visual–inertial odometry or simultaneous localization and mapping (SLAM). While historically the problem has been addressed with filtering, advancements in visual estimation suggest that nonlinear optimization offers superior accuracy, while still tractable in complexity thanks to the sparsity of the underlying problem. Taking inspiration from these findings, we formulate a rigorously probabilistic cost function that combines reprojection errors of landmarks and inertial terms. The problem is kept tractable and thus ensuring real-time operation by limiting the optimization to a bounded window of keyframes through marginalization. Keyframes may be spaced in time by arbitrary intervals, while still related by linearized inertial terms. We present evaluation results on complementary datasets recorded with our custom-built stereo visual–inertial hardware that accurately synchronizes accelerometer and gyroscope measurements with imagery. A comparison of both a stereo and monocular version of our algorithm with and without online extrinsics estimation is shown with respect to ground truth. Furthermore, we compare the performance to an implementation of a state-of-the-art stochastic cloning sliding-window filter. This competitive reference implementation performs tightly coupled filtering-based visual–inertial odometry. While our approach declaredly demands more computation, we show its superior performance in terms of accuracy

Calibration of Deformable Mirrors for Open-Loop Control

Deformable mirrors enable the control of wave fronts for the compensation of aberrations in optical systems and/or for beam scanning. Manufacturers of deformable mirrors typically provide calibration data that encode for the fabrication tolerances among the actuators and mirror segments to support open-loop control with high wave front fidelity and accuracy. We report a calibration method that enables users of the deformable mirrors to measure the response of the mirror itself to validate and improve the calibration data. For this purpose, an imaging off-axis Michelson interferometer was built that allowed measuring the mirror topography with high accuracy and sufficient spatial resolution. By calibrating each actuator over its entire range, the open-loop performance for our deformable mirror was improved

KEYFRAME-BASED VISUAL-INERTIAL SLAM USING NONLINEAR OPTIMIZATION

Abstract—The fusion of visual and inertial cues has become popular in robotics due to the complementary nature of the two sensing modalities. While most fusion strategies to date rely on filtering schemes, the visual robotics community has recently turned to non-linear optimization approaches for tasks such as visual Simultaneous Localization And Mapping (SLAM), following the discovery that this comes with significant advantages in quality of performance and computational complexity. Following this trend, we present a novel approach to tightly integrate visual measurements with readings from an Inertial Measurement Unit (IMU) in SLAM. An IMU error term is integrated with the landmark reprojection error in a fully probabilistic manner, resulting to a joint non-linear cost function to be optimized. Employing the powerful concept of ‘keyframes ’ we partially marginalize old states to maintain a bounded-sized optimization window, ensuring real-time operation. Comparing against both vision-only and loosely-coupled visual-inertial algorithms, our experiments confirm the benefits of tight fusion in terms of accuracy and robustness. I

Constraints On Porosity And Mass Loss In O-Star Winds From The Modeling Of X-Ray Emission Line Profile Shapes

We fit X-ray emission line profiles in high resolution XMM-Newton and Chandra grating spectra of the early O supergiant zeta Pup with models that include the effects of porosity in the stellar wind. We explore the effects of porosity due to both spherical and flattened clumps. We find that porosity models with flattened clumps oriented parallel to the photosphere provide poor fits to observed line shapes. However, porosity models with isotropic clumps can provide acceptable fits to observed line shapes, but only if the porosity effect is moderate. We quantify the degeneracy between porosity effects from isotropic clumps and the mass-loss rate inferred from the X-ray line shapes, and we show that only modest increases in the mass-loss rate (less than or similar to 40%) are allowed if moderate porosity effects (h(infinity) less than or similar to R-*) are assumed to be important. Large porosity lengths, and thus strong porosity effects, are ruled out regardless of assumptions about clump shape. Thus, X-ray mass-loss rate estimates are relatively insensitive to both optically thin and optically thick clumping. This supports the use of X-ray spectroscopy as a mass-loss rate calibration for bright, nearby O stars

A Generalized Porosity Formalism For Isotropic And Anisotropic Effective Opacity And Its Effects On X-Ray Line Attenuation In Clumped O Star Winds

We present a generalized formalism for treating the porosity-associated reduction in continuum opacity that occurs when individual clumps in a stochastic medium become optically thick. As in previous work, we concentrate on developing bridging laws between the limits of optically thin and thick clumps. We consider geometries resulting in either isotropic or anisotropic effective opacity, and, in addition to an idealized model in which all clumps have the same local overdensity and scale, we also treat an ensemble of clumps with optical depths set by Markovian statistics. This formalism is then applied to the specific case of boundfree absorption of X-rays in hot star winds, a process not directly affected by clumping in the optically thin limit. We find that the Markov model gives surprisingly similar results to those found previously for the single-clump model, suggesting that porous opacity is not very sensitive to details of the assumed clump distribution function. Further, an anisotropic effective opacity favours escape of X-rays emitted in the tangential direction (the venetian blind effect), resulting in a bump of higher flux close to line centre as compared to profiles computed from isotropic porosity models. We demonstrate how this characteristic line shape may be used to diagnose the clump geometry, and we confirm previous results that for optically thick clumping to significantly influence X-ray line profiles, very large porosity lengths, defined as the mean free path between clumps, are required. Moreover, we present the first X-ray line profiles computed directly from line-driven instability simulations using a 3D patch method, and find that porosity effects from such models also are very small. This further supports the view that porosity has, at most, a marginal effect on X-ray line diagnostics in O stars, and therefore that these diagnostics do indeed provide a good clumping insensitive method for deriving O star mass-loss rates

Towards bounding-box free panoptic segmentation

In this work we introduce a new bounding-box free network (BBFNet) for panoptic segmentation. Panoptic segmentation is an ideal problem for a bounding-box free approach as it already requires per-pixel semantic class labels. We use this observation to exploit class boundaries from an off-the-shelf semantic segmentation network and refine them to predict instance labels. Towards this goal BBFNet predicts coarse watershed levels and use it to detect large instance candidates where boundaries are well defined. For smaller instances, whose boundaries are less reliable, BBFNet also predicts instance centers by means of Hough voting followed by mean-shift to reliably detect small objects. A novel triplet loss network helps merging fragmented instances while refining boundary pixels. Our approach is distinct from previous works in panoptic segmentation that rely on a combination of a semantic segmentation network with a computationally costly instance segmentation network based on bounding boxes, such as Mask R-CNN, to guide the prediction of instance labels using a Mixture-of-Expert (MoE) approach. We benchmark our non-MoE method on Cityscapes and Microsoft COCO datasets and show competitive performance with other MoE based approaches while outperfroming exisiting non-proposal based approaches. We achieve this while been computationally more efficient in terms of number of parameters and FLOPs. Video results are provided here https://blog.slamcore.com/reducing-the-cost-of-understanding

Pairwise Decomposition of Image Sequences for Active Multi-View Recognition

A multi-view image sequence provides a much richer capacity for object recognition than from a single image. However, most existing solutions to multi-view recognition typically adopt hand-crafted, model-based geometric methods, which do not readily embrace recent trends in deep learning. We propose to bring Convolutional Neural Networks to generic multi-view recognition, by decomposing an image sequence into a set of image pairs, classifying each pair independently, and then learning an object classi- fier by weighting the contribution of each pair. This allows for recognition over arbitrary camera trajectories, without requiring explicit training over the potentially infinite number of camera paths and lengths. Building these pairwise relationships then naturally extends to the next-best-view problem in an active recognition framework. To achieve this, we train a second Convolutional Neural Network to map directly from an observed image to next viewpoint. Finally, we incorporate this into a trajectory optimisation task, whereby the best recognition confidence is sought for a given trajectory length. We present state-of-the-art results in both guided and unguided multi-view recognition on the ModelNet dataset, and show how our method can be used with depth images, greyscale images, or both

Simultaneous Optical Flow and Intensity Estimation from an Event Camera

Event cameras are bio-inspired vision sensors which mimic retinas to measure per-pixel intensity change rather than outputting an actual intensity image. This proposed paradigm shift away from traditional frame cameras offers significant potential advantages: namely avoiding high data rates, dynamic range limitations and motion blur. Unfortunately, however, established computer vision algorithms may not at all be applied directly to event cameras. Methods proposed so far to reconstruct images, estimate optical flow, track a camera and reconstruct a scene come with severe restrictions on the environment or on the motion of the camera, e.g. allowing only rotation. Here, we propose, to the best of our knowledge, the first algorithm to simultaneously recover the motion field and brightness image, while the camera undergoes a generic motion through any scene. Our approach employs minimisation of a cost function that contains the asynchronous event data as well as spatial and temporal regularisation within a sliding window time interval. Our implementation relies on GPU optimisation and runs in near real-time. In a series of examples, we demonstrate the successful operation of our framework, including in situations where conventional cameras suffer from dynamic range limitations and motion blur

Learning to complete object shapes for object-level mapping in dynamic scenes

In this paper, we propose a novel object-level mapping system that can simultaneously segment, track, and reconstruct objects in dynamic scenes. It can further predict and complete their full geometries by conditioning on reconstructions from depth inputs and a category-level shape prior with the aim that completed object geometry leads to better object reconstruction and tracking accuracy. For each incoming RGB-D frame, we perform instance segmentation to detect objects and build data associations between the detection and the existing object maps. A new object map will be created for each unmatched detection. For each matched object, we jointly optimise its pose and latent geometry representations using geometric residual and differential rendering residual towards its shape prior and completed geometry. Our approach shows better tracking and reconstruction performance compared to methods using traditional volumetric mapping or learned shape prior approaches. We evaluate its effectiveness by quantitatively and qualitatively testing it in both synthetic and real-world sequences

In-place scene labelling and understanding with implicit scene representation

Semantic labelling is highly correlated with geometry and radiance reconstruction, as scene entities with similar shape and appearance are more likely to come from similar classes. Recent implicit neural reconstruction techniques are appealing as they do not require prior training data, but the same fully self-supervised approach is not possible for semantics because labels are human-defined properties.We extend neural radiance fields (NeRF) to jointly encode semantics with appearance and geometry, so that complete and accurate 2D semantic labels can be achieved using a small amount of in-place annotations specific to the scene. The intrinsic multi-view consistency and smoothness of NeRF benefit semantics by enabling sparse labels to efficiently propagate. We show the benefit of this approach when labels are either sparse or very noisy in room-scale scenes. We demonstrate its advantageous properties in various interesting applications such as an efficient scene labelling tool, novel semantic view synthesis, label denoising, super-resolution, label interpolation and multi-view semantic label fusion in visual semantic mapping systems